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Mike  Gagliardi  has  more  than  25  years  experience  in  real-time,  mission- 
critical  software  architecture  and  engineering  activities  on  a  variety  of  DoD 
systems.  He  currently  works  in  the  SEI  Research,  Technology,  and  System 
Solutions  Program  on  the  Architecture-Centric  Engineering  Initiative,  and  is 
leading  the  development  of  architecture  evaluation  and  quality  attribute 
specification  methods  for  system  and  SoS  architectures. 


Leveraging  Our  Success  in  Software  Architecture 


We  have  lots  of  experience  and  success  with  proven  methods  for  quality 
attribute  elicitation  (QAW*)  and  architecture  evaluation  (ATAM**)  of 
software  architectures,  in  many  contexts: 

•  DoD 

•  Commercial 

•  Acquisition 

These  methods  have  been  adopted  by  a  wide  variety  of  organizations  to 
specify  quality  attributes  and  identify  architectural  risks  early  in  the  life- 
cycle. 

We  have  expanded  the  scope  to  system  and  system-of-system  (SoS) 
architectures. 
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SEI  Software  Architecture  Axioms 


1 .  Software  architecture  a  bridge  between  business  and  mission  goals 
and  a  software-reliant  system. 

2.  Quality  attribute  requirements  drive  the  design  of  the  software 
architecture. 

—  Quality  attribute  requirements  stem  from  business  and  mission  goals. 

—  Key  quality  attributes  need  to  be  characterized  in  a  system-specific  way. 

—  Scenarios  are  a  powerful  way  to  characterize  quality  attributes  and  represent 
stakeholder  views. 

3.  Software  architecture  drives  software  development  throughout  the 
life  cycle. 

—  Software  architecture  must  be  central  to  software  development  activities. 

—  These  activities  must  have  an  explicit  focus  on  quality  attributes. 

—  These  activities  must  directly  involve  stakeholders  -  not  just  the  architecture 
team. 
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Conceptual  Flow  of  System  and  Software  ATAM 
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Concerns  Addressed  by  Software  Architecture 


Achieving  Key  Properties  in  Software-Reliant  Systems 

Design-time 

Run-time 

Software  In  Its 
Environment 

•  Modifiability 

•  Performance 

•  Maintainability 

•  Security 

•  Usability 

•  Reusability 

•  Reliability 

•  Supportability 

•  Portability 

•  Availability 

•  Configurability 

•  Testability 

•  Scalability 

•  Sustainability 

•  Etc. 

•  Interoperability 

•  Buildability 

•  Throughput 

•  Capacity 

•  Etc. 

•  Etc. 
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Problem 


Integration  and  operational  problems  arise  due  to  inconsistencies, 
ambiguities,  and  omissions  in  addressing  quality  attributes  between 
system  and  software  architectures.  This  is  further  exacerbated  in  an 
SoS. 

•Example  quality  attributes:  predictability  in  performance,  security,  availability/reliability, 
usability,  testability,  safety,  interoperability,  maintainability,  force  modularity,  spectrum 
management. 


Functionality  and  capability  are  important,  but  the  architecture 
must  be  driven  by  the  quality  attributes.  Identifying  and  addressing 
quality  attributes  early  and  evaluating  the  architecture  to  identify 
risks  is  key  to  success. 

Architecture  plays  an  important  role  in  every  stage. 
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Common  Symptoms  Stemming  From 
Architectural  Deficiencies 


Operational 

•  Communication  bottlenecks  under  various  load  conditions  in  systems  or 
throughout  system  of  systems 

•  Systems  that  hang  up  or  crash;  portions  that  need  rebooting  too  often 

•  Difficulty  synching  up  after  periods  of  disconnect  and  resume  operations 

•  Judgment  by  users  that  system  is  unusable  for  variety  of  reasons 

•  Database  access  sluggish  and  unpredictable 


Developmental 

•  Integration  schedule  blown,  difficulty  identifying  root  causes  of  problems 

•  Proliferation  of  patches  and  workarounds  during  integration  and  test 

•  Integration  of  new  capabilities  taking  longer  than  expected,  triggering  breaking 
points  for  various  resources 

•  Significant  operational  problems  ensuing  despite  passage  of  integration  and  test 

•  Anticipated  reuse  benefits  not  being  realized 
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The  Need  for  Augmented  Mission  Threads  in 
DoD  SoS  Architecture  Definition 


DoDAF  is  the  SoS  architecture  framework  for  the  DoD.  It  provides  a 
good  set  of  architectural  views  for  an  SoS  architecture.  However,  it 
inadequately  addresses  cross-cutting  quality  attribute  considerations. 

System  use  cases  focus  on  a  functional  slice  of  the  system. 


More  than  DoDAF  and  system  use  cases  are  needed  to  ensure  that 
the  SoS  architecture  satisfies  its  cross-cutting  quality  attribute  needs. 


SoS  end-to-end  mission  threads  augmented  with  quality  attribute 
considerations  are  needed  to  help  define  the  SoS  Architecture 
precepts  and  guidelines,  and  then  later  evaluate  the  SoS  architecture. 
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Definitions  (DoD  Context) 


Vignette:  A  description  of  the  geography,  own  force  structure  and 
mission,  strategies  and  tactics,  the  enemy  forces  and  their  attack 
strategies  and  tactics,  including  timing.  There  may  be  associated 
Measures  of  Performance  (MOP)  and  Measures  of  Effectiveness 
(MOE).  A  vignette  provides  context  for  one  or  more  mission  threads. 

Mission  Thread: 

A  sequence  of  end-to-end  activities  and  events  beginning  with  an 
opportunity  to  detect  a  threat  or  element  that  ought  to  be  attacked  and 
ending  with  a  commander’s  assessment  of  damage  after  an  attack. 
C4ISR  for  Future  Naval  Strike  (Operational) 

Sustainment:  A  sequence  of  activities  and  events  which  focus  on 
development,  deployment  and  maintenance. 
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Vignettes  Are  the  Starting  Point  -  Example 
Wording 


Two  ships  (Alpha  and  Beta)  are  assigned  to  integrated  air  and  missile 
defense  (IAMD)  to  protect  a  fleet  containing  two  high-value  assets 
(HVA).  A  surveillance  aircraft  SA  and  4  UAVs  are  assigned  to  the  fleet 
and  controlled  by  the  ships.  Two  UAVs  flying  as  a  constellation  can 
provide  fire-control  quality  tracks  directly  to  the  two  ships.  A  three¬ 
pronged  attack  on  the  fleet  occurs: 

•  20  land-based  ballistic  missiles  from  the  east 

•  5  minutes  later  from  5  aircraft-launched  missiles  from  the  south 

•  3  minutes  later  from  7  submarine-launched  missiles  from  the  west. 

The  fleet  is  protected  with  no  battle  damage. 
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Vignettes  Are  the  Starting  Point  -  Example 
Context 
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Mission  Threads  Flow  from  Vignettes  -  Example 
(Non-Augmented) 


1 . 20  land-based  missiles  launched  -  X  minute  window 

2.  Satellite  detects  missiles  -  cues  CMDR 

3.  CMDR  executes  re-planning  -  reassigns  Alpha  and  Beta 

4.  Satellite  sends  track/target  data  -  before  they  cross  horizon 

5.  Ships’  radars  are  focused  on  horizon  crossing  points 

N  Engagement  cycle  is  started  on  each  ship 
N+1 .  Aircraft  are  detected  heading  for  fleet 
N+2.  SA  detects  missile  launches  -  tells  CMDR 
N+3.  CMDR  does  re-planning  -  UAVs  are  re-directed 
N+4.  FCQ  tracks  are  developed  from  UAV  inputs 
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Other  End-to-End  Mission  Threads 


have  also  proven  useful  in 
commercial  SoS  contexts, 
we  have  piloted  this  in: 

•  Commercial  Call-Center  context 

•  Stock  Market  Transaction  context 


The  methods  hold  up, 
the  inputs  change: 

•  End-to-End  Business  Process 
Threads 

•  End-to-End  Transaction  Threads 


Internet 
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Overview 
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Mission  Thread  Workshop 
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Mission  Thread  Workshop  (MTW)  Purpose 


The  MTW  augments  SoS  mission  threads  with  quality  attribute 
considerations  that  shape  the  SoS  architecture  and  identifies  SoS 
architectural  challenges,  as  early  in  the  SoS  development  cycle 
as  possible. 

The  mission  thread  augmentation  is  performed  with  inputs  from  key 
SoS  stakeholders  and  is  facilitated  by  the  SEI. 

The  augmented  mission  threads  and  challenges  are  used  to 
develop  the  SoS  architecture  and  then  later  to  evaluate  the  SoS 
architecture. 

There  will  be  a  series  of  MTWs  depending  on  scope,  scale,  and 
schedule  considerations. 
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MTW  sequence  planning/scheduling  and 
vignette  and  MT  development/selection 

Criteria  for  development/selection  of  vignettes  and  MTs 

•  Capability  Coverage 

•  New  requirements/capabilities 

•  Stressing  the  SoS 

•  constituent  systems,  communications,  etc 

•  New  integrated  existing  capabilities 

You  can  only  do  so  many  of  these...  make  them  count. 
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Preparation 


The  SoS  Program  Manager  develops  a  overview  presentation  on  the 
SoS  Mission  /  Business  Drivers  (see  SoS  Mission  /  Business  driver 
presentation  template). 

The  SoS  Architect  develops  an  overview  presentation  on  the  SoS 
Architecture  Plans  (see  SoS  Architecture  Plans  presentation 
template). 

The  SEI  meets  with  the  SoS  Architect  and  PM  to: 

•  Determine  if  the  vignettes  and  MTs  are  sufficient  to  proceed. 

•  Provide  feedback  on  the  two  presentations 

•  Reach  agreement  on  scope  and  series  of  MTWs 

•  Identify  Stakeholders 

•  Determine  logistics 
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Stakeholders  are  Key! 


When  developing  the  initial  set  of  vignettes  and  MTs,  it  is  critical  to 
associate  them  with  the  key  stakeholder  types  that  will  be 
necessary  to  participate  in  the  Workshops. 


There  may  be  groups  of  stakeholder  types  that  are  not  necessary  for 
specific  vignettes. 


Example  stakeholders:  (leads  in  the  following) 
Modeling  and  Simulations 

•  Integration  and  Test  Facility  (SIL) 

•  CONOPS,  DRM,  Operational  Analysts, 

•  SoS,  System  and  Software  Architects 

•  Legacy  System  Architects 
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MTW  Inputs  - 1 


SoS  Business  and  Mission  Drivers  Presentation  (15  mins) 

•  A  representative  from  the  SoS  stakeholder  community  presents  the 
SoS  business  and/or  mission  drivers  including  the 
business/programmatic  context,  high-level  functional  requirements, 
high-level  constraints,  high-level  quality  attributes,  acquisition  strategy, 
etc. 


SoS  Architecture  Plans  Presentation  (30  mins) 

•  The  SoS  architect  presents  the  architecture  development  plans 
including  key  business/programmatic  requirements,  key  technical 
requirements  and  constraints  that  will  drive  architectural  decisions,  any 
relevant  existing  context  diagrams,  high-level  SoS  diagrams  and 
descriptions,  development  spirals  and  integration  schedule. 
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MTW  Inputs  -  2 


Vignettes 

•  A  description  of  the  geography,  own  force  structure  and  mission, 
strategies  and  tactics,  the  enemy  forces  and  their  attack  strategies  and 
tactics,  including  timing.  There  may  be  associated  Measures  of 
Performance  (MOP)  and  Measures  of  Effectiveness  (MOE). 

—  An  SoS  will  typically  support  multiple  vignettes,  i.e.  multiple  mission 
areas  such  as  Air  Defense,  Ballistic  Missile  Defense, 
Replenishment,  Mobility,  etc. 

—  Each  vignette  typically  supports  multiple  mission  threads 

Mission  Threads,  types: 

•  Operational  -  A  sequence  of  activities  and  events  beginning  with  an 
opportunity  to  detect  a  threat  or  element  that  ought  to  be  attacked  and 
ending  with  a  commander’s  assessment  of  damage  after  an  attack. 

•  Sustainment:  A  sequence  of  activities  and  events  which  focus  on 
development,  deployment  and  maintenance. 
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Typical  MTW  Agenda 


08:00-08:15 

08:15-08:45 

08:45-09:00 

09:00-09:40 

09:40-09:55 

09:55-12:00 

12:00-13:00 

13:00-13:20 

13:20-15:00 

15:00-15:15 

15:15-15:45 

15:45-17:00 


Welcome/Introductions/Opening  Remarks  (joint) 

MTW  Overview  (SEI) 

Business  Drivers  and  Quality  Attributes  (Architect) 

OV-1  &  Vignettes  Overview  (Architect) 

Break 

Augmentation  of  1st  mission  thread  (SEI  facilitated) 

Lunch 

Review  OV-1  and  vignette  associated  with  2nd  mission  thread  (Architect) 
Augmentation  of  2nd  mission  thread  (SEI  facilitated) 

Break 

Review  OV-1  and  vignette  associated  with  3rd  mission  thread  (Architect) 
Augmentation  of  3rd  mission  thread  (SEI  facilitated) 
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Augmentation  Process  -  Per  Mission  Thread 


1)  For  each  event  in  the  mission  thread: 

•  Elicit  quality  attribute  considerations.  Capturing  any  engineering  issues,  assumptions, 
challenges,  additional  use  case  and  mission  threads  (with  QA  context  etc.) 

•  Capture  any  capability  and/or  mission  issues  that  arise. 

2)  Elicit  any  over-arching  quality  attribute  considerations 

•  Capturing  any  over-arching  assumptions,  engineering  issues,  challenges,  additional 
use  cases  and  mission  threads  (with  QA  context)  etc. 

3)  Capture  any  capability  and/or  mission  issues  that  arise. 

4)  Capture  any  MT  extensions  for  a  later  pass. 

Parking  Lot  -  for  organization,  programmatic,  non-technical  issues  that  arise  (will 
not  be  further  pursued  in  the  MTW). 


SEI  facilitates  and  scribes  using  a  pre-defined  MTW  template. 

Stakeholder  Inputs  are  Key. 
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Rules 


SEI  will  provide  the  facilitation  and  scribing. 

This  is  a  big  crowd:  side  conversations,  cell  calls,  etc.  will  not  be 
allowed  to  disrupt  the  meeting. 

Once  an  issue  is  identified  and  discussed,  we  will  not  allow  it  to  be  re¬ 
discussed.  It  will  be  noted  at  the  appropriate  place. 

Will  keep  the  discussions  within  scope. 

Will  not  get  into  the  details  of  potential  solutions  to  issues. 

Programmatic,  organizational,  and  other  non-technical  issues  will  be 
noted,  but  not  discussed  in  detail. 
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Example  MTW  Walk-Through 


At  this  point,  we  will  switch  to  the  MTW  template  which  is  partially 
filled  in.  We  will  walk  through  the  MTW  augmentation  process 
using  the  DoD  SoS  example. 
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MTW  Outputs 


Individual  MTWs 

Augmented  Mission  Threads 

•  Over-arching  quality  attribute  augmentations  for  the  mission  thread 

•  Capability  and  mission  augmentations  to  the  mission  thread 

•  Quality  attribute  augmentations  for  each  event  in  the  mission  thread 

•  Identified  mission/additional  use  cases  (with  context)  and  mission  threads 

Challenges 

•  Architectural,  capability  and  mission  challenges  derived  from  the  mission  thread  augmentations. 

•  The  MTW  team  will  roll  up  challenges  from  the  data  and  provide  an  out-brief  of  the  challenges. 

•  Mapped  to  contributing  augmented  mission  thread  steps 

•  These  are  vetted  and  updated  with  the  principals 

•  Identify  any  candidate  legacy  system  architecture  that  may  require  architecture  evaluation. 


SoS  Architectural  Challenges 

Report  upon  completion  of  series  of  MTWs: 

•  SoS  architectural  challenges  derived  and  roiled  up  from  the  mission  thread  augmentations; 
upon  completion  of  the  series  of  mission  thread  workshops  for  the  SoS. 

•  Meet  with  the  principals  to  “rack  and  stack”  challenges. 
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Examples  of  Rolled-up  Challenges 


End-End  resource  management  strategy  needs  developed;  esp. 
regarding  issues  dealing  with  supporting  the  number  of  missiles 
and  radar  coverage. 

Fault  model  and  recovery  activities  needs  further  definition  and 
architectural  guidance  needs  developed 

Degraded  modes  of  operation  strategy  and  associated  architectural 
support  needs  developed. 

Performance  timelines  and  deadlines  need  defined  and  decomposed 

Manning/automation  studies/analyses  insufficient 

Sensor  coordination  between  the  two  ships  and  the  UAVs  needs 
further  analysis. 
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MTW  Experiences  - 1 


Conducted  a  total  of  22  MTWs  (over  60  mission  threads  augmented), 
each  MTW  is  a  1 .5  day  meeting 

Plan  4  MTs  per  MTW,  but  expect  to  augment  3. 

Expect  25-30  stakeholders  to  want  to  participate  per  MTW.  Benefits 
from  strong  facilitation  and  independent  3rd  party  leadership. 

Clients  developed  very  good  first  pass  vignettes  and  MTs  after  initial 
introduction. 

Criteria  for  MT  selection  include:  New  capability,  High  perceived  risk, 
proposal  differentiators,  etc. 

DoDAF  OV-Ts  were  sufficient  level  of  documentation  going  into  the 
MTWs 
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MTW  Experiences  -  2 


DoDAF  OV-l’s  were  sufficient  level  of  documentation  going  into  the 
MTWs 

Mission  thread  step  elaboration  focused  on: 

•  Command  authority,  network  communications,  step  constraints 

•  Manned  vs  Automated,  timelines,  planning  considerations 

•  Availability  and  Survivability  considerations 

•  Readiness,  environmental  conditions,  start  up/shut  down 

•  Current  capabilities/extensions 

•  CONOPS  considerations 

•  Assumption  clarifications  and  issues 
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MTW  Experiences  -  3 


Quality  Attributes  Considerations: 

•  Timeline  decomposition  often  built  into  thread  (weeks  to  seconds) 

•  Availability/  Degraded  Operation  /  Resource  Management  under¬ 
developed 

•  Focus  on  operational  MTs,  separate  MTW  for  development  and  support 

•  Over-arching  MT  pass  collects  much  of  the  QA  considerations 

•  Identified  additional  use  cases  and  MTs  (e.g.  survivability) 

Challenges: 

•  Some  challenges  need  to  be  kicked  up  to  the  SoS  architecture  level  to 
address,  while  others  need  to  be  addressed  by  systems  engineering 

•  Drives  an  SoS  Architecture  and  Guidelines  Document 
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MTW  -  Initial  Results  - 1 


The  MTW  and  SoS  Arch  Evaluation  methods  adopted  by  a  Navy  SoS 
organization  and  required  in  their  architecture  development 
process 

Many  of  the  identified  challenges  drove  early  risk  mitigation  activities 
(e.g.  prototyping,  EDM,  white  papers,  modeling  and  sim). 

Many  new  use  cases  and  additional  mission  threads  identified.  The 
QA  considerations  will  be  included  in  the  use  cases. 

Excellent  vehicle  to  promote  communication  between  architects  and 
stakeholders. 

Capability  and  Mission  Challenges  were  identified  as  well  as 
Architectural  Challenges. 
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MTW  -  Initial  Results  -  2 


SoS  Architecture  and  Guidelines  document  is  needed.  Developed  a 
template  for  use  on  Army  and  Navy  SoS  Programs. 

Supports  programs’  DoDAF  architecture  development  efforts. 
Normalized  the  OV-ls  and  informed  and  drove  many  subsequent 
DoDAF  views  (e.g.  OV-5,  OV-2,  OV-3,  OV-4,  OV-6c,  SV-5a,  SV-4a, 
SV-1,  SV-3) 

3rd  Party  facilitation  by  the  MTW  facilitators  enabled  the  leads  to  think 
about  and  participate  in  the  discussions  rather  than  trying  to 
lead/control  the  meetings 

Method  worked  for  non-software  elements,  as  well  as  software¬ 
intensive  elements 
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Legacy  System  Architecture  Evaluation 
Using  ATAM 


Software  Engineering  Institute  Carnegie  Mellon 


SEI  Webinar 

Gagliardi,  Wood,  Morrow,  Klein 
©2010  Carnegie  Mellon  University 


Legacy  System  Architecture  Evaluation  -  Early 

•  Early  elicitation  of  quality  attribute  considerations 

•  Early  identification  and  addressing  of  architecture  challenges  (e.g.  candidate  legacy 
system  architecture  evaluation) 

•  Early  identification  and  mitigation  of  architectural  risks 


SoS  Business  /  Mission  Drivers 


Warfare  Vignettes 
Mission  Threads 
SoS  Architecture  Plans 


Mission 

Thread 

Workshop 


System  ATAM 
on  candidate 
legacy  system 


Augmented  Mission  Threads 
SoS  Architecture  Challenges 


SoS 

Architecture 

Evaluation 


Sys  Arch  Risks 


V  V 


SoS  Architecture  Risks 

Problematic  systems 
identified  with  the 
augmented  mission 
threads 


SoS  Architecture 
System  Architectures 
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SoS  and  System  Architecture(s)  Acquisition  /  Development 
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Purpose  of  the  System  ATAM 


The  System  ATAM  is  a  method  that  helps  stakeholders  ask  the  right 
questions  to  discover  potentially  problematic  architectural  decisions. 


Purpose  is  to  assess  the  consequences  of  system  and  software 
architectural  decisions  in  light  of  quality  attribute  requirements  and 
business  goals;  and  to  identify  architectural  risks. 

The  purpose  is  NOT  to  provide  precise  analyses;  the  purpose  IS  to 
discover  risks  created  by  architectural  decisions. 


Discovered  risks  can  then  be  made  the  focus  of  mitigation  activities. 
Tradeoffs  can  be  explicitly  identified  and  documented 
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Using  the  augmented  mission  threads  to  seed 
the  system  architecture  evaluation 


Comments  from  augmented  mission  thread: 

The  Defensive  Engagement  System  may  not  be  able  to  support  the  deconfliction 
timeline  for  5  incoming  missiles. 

The  Defensive  Engagement  System  may  not  have  the  capability  to  acknowledge 
Beta’s  acceptance  of  its  assignment  of  2  missiles. 

Is  the  Defensive  Engagement  System  capable  of  sending  track  updates  to  the 
interceptor  missiles  that  Beta  had  launched  within  the  intercept  timeline? 


In  preparation,  the  System  ATAM  lead  meets  with  SoS  and  appropriate  system 
architects  to  discuss  what  is  in  and  out  of  scope  concerning  the  system 
under  analysis  and  if  appropriate  documentation  exists. 

Agreement  is  reached  on  the  scenarios  )based  upon  the  augmented  mission 
threads)  with  the  understanding  that  additional  scenarios  can  be  added 
during  the  legacy  system  architecture  evaluation. 
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Conceptual  Flow  of  System  ATAM  Variant  for 
Legacy  Systems  I  AUg^ied 

w  #  #  ^  Mission  Threads  and 

Use  Cases  based  on  MTWs 


System  and 
Software 
Architecture 


Impacts 
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tradeoffs 
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Examples  of  Scenarios 


Scenarios  address  both  system  and  software  aspects.  Consist 
of  Stimulus,  Environment  and  Response. 

Growth  scenarios 

The  Defensive  Engagement  System  (DES)  is  able  to  support  de- 
con  f I iction  of  7  incoming  missiles  using  own -ship  and  external 
information  within  5  seconds. 

An  upgraded  DES  is  able  to  reduce  the  confliction  time  by  40%  of  7 
incoming  missiles  with  no  loss  of  existing  functionality. 

Exploratory  scenario 

The  DES  is  able  to  operate  at  up  to  80%  of  its  time  budget  for  de- 
conf I  iction  of  7  incoming  missiles  with  8  coalition  UA  Vs  and  3  coalition 
helicopters  operating  in  its  vicinity. 
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Stakeholders  and  Evaluators 


Stakeholders  will  consist  of: 

System  Architects  of  relevant,  associated  systems  to  system  under 
evaluation 

SoS  Architects  who  know  the  total  system  and  how  the  system  under 
evaluation  is  envisioned  to  fit  in 

Relevant  stakeholders  of  the  system  under  evaluation  in  the  areas  of 
requirements,  development,  T&E,  sustainment,  M&S 


ATAM  evaluators  will  look  to  identify/expose  potential  system  and 
software  architecture  risks,  with  the  help  of  the  stakeholders. 
Subject  matter  experts  may  be  used  on  the  evaluation  team,  if 
necessary. 
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Walk-through  of  a  scenario  derived  from 
augmented  MT 


The  Defensive  Engagement  System  (DES)  is  able  to  support  de- 
confliction  of  7  incoming  missiles  using  own-ship  and  external 
information  within  5  seconds. 

•  System  architect  identifies  that  currently  DES  can  support  3  incoming 
missiles  with  25%  spare  capacity  given  the  existing  hardware. 

•  The  software  architect  reveals  that  the  system  has  a  monolithic 
software  architecture  which  is  tightly  coupled  to  the  existing  hardware. 

•  The  architect  identifies  that  upgraded  hardware  is  available  for  the 
system  which  will  provide  the  needed  performance  upgrade,  but  the 
software  will  need  to  be  re-designed  to  take  advantage  of  the  upgrade. 


SoS  and  DES  architects  and  managers  negotiate  how  to  proceed 
based  on  architectural  risks  identified  and  associated  risk 
mitigation  options. 
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SoS  Architecture  Evaluation 
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SoS  Architecture  Evaluation  Purpose 


The  SoS  Architecture  Evaluations  identifies  SoS 
architectural  risks  by  probing  the  SoS  architecture,  using 
the  augmented  SoS  mission  threads  and  challenges,  to 
evaluate  the  SoS  architecture.  It  also  identifies  any 
problematic  systems  that  require  further  evaluation. 


There  will  be  a  series  of  SoS  Architecture  Evaluations 
depending  on  scope,  scale,  and  schedule  considerations. 
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Stage  1 :  Preparation  - 1 


Review  results  of  MTW,  noting  the  architectural  challenges  and  expected 
resolutions;  and  highlight  augmentations  that  require  further 
explanation 

Identify  the  mission  threads  for  the  SoS  Arch  Eval  with  the  SoS  architect 
•  Assume  that  only  1  -2  mission  threads  can  be  evaluated  per  day  max. 

Develop  and  review  the  SoS  business/mission  drivers  and  the  SoS  and 
System/SW  architecture  presentations 

Review  SoS  and  system  architecture  documentation  for  sufficiency 

Identify  stakeholders  (some  to  assist  with  the  evaluation) 
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Stage  1 :  Preparation  -  2 


Develop  a  schedule  of  the  evaluations 

Set  up  logistics  and  send  out  read-ahead  with  invitations 

Walk-through  one  mission  thread  for  practice 

Identify  evaluation  team 

•  Lead,  Scribe,  3  Evaluators 

—  ATAM  evaluator  qualified 

•  Domain  SMEs  (e.g.  Communications,  sensors,  weapons,  platforms, 
warfare  experts) 

Evaluation  team  reviews  the  inputs  and  becomes  familiar  with  the  SoS 
Architecture  in  advance  of  the  evaluation 
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Stage  2:  Execution  - 1 


Note:  2  day  max  for  each  SoS  Arch  Eval 

•  Probably  will  only  get  through  2  mission  threads 

Presentations: 

•  SoS  Business/Mission  Driver  Presentation 

•  SoS  Architecture  Presentation 

•  Augmented  Mission  Threads  for  this  evaluation 

•  Architectural  Challenges  from  the  MTW 
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Stage  2:  Execution  -  2 


Analysis  for  each  architecture  challenge 

•  The  architect  describes  how  the  architecture  satisfies  each 
architecture  challenge  indentified  in  the  MTWs 

Analysis  for  each  augmented  mission  thread 

•  Start  with  SoS  Architect 

•  Walkthrough  the  documented  architecture,  describing  how  the 
architecture  satisfies  the  MT 

—  Step  by  step  probing  all  highlighted  QAs,  looking  for  risks 
—  Some  hybrid  of  completing  a  step  for  all  QAs  and  completing  all  steps  for 
a  QA. 


For  each  analysis  above: 

•  SoS  architect  can  hand  over  to  system  and  s/w  architects  as  needed 

•  The  evaluation  team  probes  for  risks 

•  Scribe  risks,  non-risks  and  issues,  etc  using  the  evaluation  template 
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Stage  2:  Execution  -  3 


Strong  facilitation  to  stay  on  track;  Do  not  go  too  deep  in 
system  architectures,  whatever  is  architecturally  significant 
for  the  MT  at  the  SoS  level. 

Create  “Parking  Lot”  for  non-technical  issues 

Summarize  findings  in  an  out-brief 
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Stage  3:  Roll-up  and  Follow-up 


At  the  end  of  each  SoS  Arch  Eval: 

•  Output  Briefing 

•  SoS  Architectural  Risk  Themes,  Non  risks,  Trade-offs 

•  Any  non-architectural  issues  discovered 

•  One  example  of  an  mission  thread  analysis  with  discovered  SoS 
architectural  risks,  trade-off  points  and  non-risks 

•  Any  problematic  systems  identified  for  future 

•  Identify  “parking  lot”  issues 

•  Summary  Report  of  individual  SoS  Arch  Eval 

•  Detailed  write-ups  on  the  risk  themes,  non-risks,  etc  found  during 
the  evaluation 

•  Summary  of  the  SoS  architecture,  approaches,  guidelines,  etc 

•  Summary  of  the  SoS  business  and  mission  drivers,  quality 
attributes,  summarizing  implications  of  any  mismatches  between 
SoS  and  systems 
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SoS  Arch  Evals  Roll-up 


At  the  end  of  the  series  of  SoS  Arch  Evals 

•  Evaluation  team  meets  to  roll-up  the  findings  from  the  series  of  SoS 
Arch  Evals 

•  Annotated  Summary  Briefing 

•  SoS  Architectural  Risk  Themes  and  Non-risks  (rolled  up) 

•  Any  non-architectural  issues  discovered  (rolled  up) 

•  Identify  problematic  areas  and  schedule  “focused”  architecture 
evaluations  (e.g.  System  &  Software  ATAM) 

•  Recommendations 

•  SoS  Arch  Eval  Summary  Report 

•  Detailed  write-ups  on  the  risk  themes,  non-risks,  etc  found  during 
the  evaluation 

•  Summary  of  the  SoS  architecture,  approaches,  guidelines,  etc 

•  Summary  of  the  SoS  business  and  mission  drivers,  quality 
attributes,  summarizing  implications  of  any  mismatches  between 
SoS  and  systems 

•  Recommended  Next  Steps 
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Value  to  Customers  at  Various  Stages  - 1 


Working  early  with  the  Program  Office  to  develop  the  proper 
architecture-centric  acquisition  strategy  and  associated  language  for 
proposals,  contracts,  etc.  will 

•  drive  the  contractors  to  do  the  right  thing  architecturally  early 

•  provide  visibility  to  the  program  office  into  the  architecture’s  goodness 

•  identify  architectural  risks  early 


This  is  the  biggest  point  of  leverage  within  DoD  programs.  We  have 
demonstrated  its  effectiveness  on  DoD  programs  in  software 
architecture.  Our  many  pilots  indicate  that  this  is  true  for  SoS  and 
system  architecture  as  well. 


Software  Engineering  Institute  Carnegie  Mellon 


SEI  Webinar 

Gagliardi,  Wood,  Morrow,  Klein 
©2010  Carnegie  Mellon  University 


55 


Value  to  Customers  at  Various  Stages  -  2 


MTW  -  Early  elicitation  of  SoS  quality  attribute  needs,  architectural 
challenges,  mission  and  capability  challenges,  system  use  cases. 

•  Stakeholder-elicited  quality  attribute  information  available  to  the  SoS 
architecture  developers  (and  integrators  and  testers);  also  used  to 
inform  the  system  and  software  architecture  development/acquisition 
activities. 


•  Challenges  are  identified  early  in  the  life  cycle,  to  prevent  them  from 
becoming  risks  later. 

Architecture  Evaluation  -  Early  identification  of  SoS  architecture  risks 
and  problematic  constituent  systems. 


•  The  architects,  along  with  the  program  office,  can  identify,  prioritize, 
and  mitigate  risks  early  in  the  life  cycle,  prior  to  integration. 


•  Addressing  the  risks  prior  to  integration  will  reduce  integration  and 
operational  risks. 
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Contact  Information 


Mike  Gagliardi  -  miq@sei  cmu.edu 
Software  Engineering  Institute 
Carnegie  Mellon  University 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213 
412-268-7738 
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Definitions  - 1 


A  System  of  Systems  is  “a  set  or  arrangement  of  systems  that  results 
when  independent  and  useful  systems  are  integrated  into  a  larger 
system  that  delivers  unique  capabilities.”  [OSD  Systems  Engineering 
Guide  for  Systems  of  Systems,  August  2008] 


OSD  SE  Guide  defines  four  types  of  SoSs: 

Directed 

Acknowledged 

Collaborative 

Virtual 

The  tutorial  will  be  addressing  Directed  and  Acknowledged  SoSs 
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Definitions  -  2 


Directed.  Directed  SoS  are  those  in  which  the  integrated  system-of-systems  is  built  and  managed  to 
fulfill  specific  purposes.  It  is  centrally  managed  during  long-term  operation  to  continue  to  fulfill 
those  purposes  as  well  as  any  new  ones  the  system  owners  might  wish  to  address.  The 
component  systems  maintain  an  ability  to  operate  independently,  but  their  normal  operational 
mode  is  subordinated  to  the  central  managed  purpose. 

Acknowledged.  Acknowledged  SoS  have  recognized  objectives,  a  designated  manager,  and 
resources  for  the  SoS;  however,  the  constituent  systems  retain  their  independent  ownership, 
objectives,  funding,  and  development  and  sustainment  approaches.  Changes  in  the  systems  are 
based  on  collaboration  between  the  SoS  and  the  system. 

Collaborative.  In  collaborative  SoS  the  component  systems  interact  more  or  less  voluntarily  to  fulfill 
agreed  upon  central  purposes.  The  Internet  is  a  collaborative  system.  The  Internet  Engineering 
Task  Force  works  out  standards  but  has  no  power  to  enforce  them.  The  central  players 
collectively  decide  how  to  provide  or  deny  service,  thereby  providing  some  means  of  enforcing 
and  maintaining  standards. 

Virtual.  Virtual  SoS  lack  a  central  management  authority  and  a  centrally  agreed  upon  purpose  for  the 
system-of-systems.  Large-scale  behavior  emerges — and  may  be  desirable — but  this  type  of  SoS 
must  rely  upon  relatively  invisible  mechanisms  to  maintain  it. 
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Definitions  -  3 


An  Architecture  is  the  structure  of  components,  their  relationships,  and 
the  principles  and  guidelines  governing  their  design  evolution  over 
time  [IEEE  Std  610.12  and  DoDAF]. 


An  SoS  Architecture  is  the  structure  of  constituent  systems,  their 
relationships,  and  the  principles  and  guidelines  governing  their 
design  evolution  over  time. 


Need  to  elaborate  on  this  to  clarify. 
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Elaboration 


The  structure(s)  of  the  constituent  systems  include: 

—  Allocation  of  functionality  to  each  constituent  system 

—  End-to-end  activity  flows  and  communications,  including  operational, 
sustainment,  development,  and  deployment  activities. 

—  Externally  visible  properties  and  interfaces  of  the  constituent  systems, 
including  behaviors,  dependencies,  use  of  shared  resources,  etc. 

—  Relationship  among  organizational  entities  and  the  constituent  systems  at 
each  phase  of  the  SoS  lifecycle. 

—  Rationale  and  governance  policies,  for  example,  criteria  for  decisions 
about  constituent  system  inclusion,  continued  participation  and 
termination. 


Depending  on  the  type  of  SoS: 

—  the  point  at  which  the  structures  are  determined  and  by  whom  can  vary 

—  the  level  of  specificity  and  abstractions  can  vary 
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Methods/Activities  Superimposed  Over  DoD 
SoS  Life-Cycle 
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Material  Solution  Analysis  Phase 
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Technology  Development  Phase 
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